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Abstract 


Classification  of  documents  involves  three  distinct  major  pro¬ 
cesses,  The  first  two  processes  of  defining  a  structure  of  categories 
and  determining  a  basis  for  the  classification  decision  are  usually 
performed  by  a  classificationist,  while  the  third  process  of  classify¬ 
ing  documents  into  categories  is  performed  by  a  classifier.  The  ob¬ 
jectives  of  our  approach  is  to  develop  computer  techniques  to  perform 
the  second  and  third  processes. 

Previous  experiments  indicate  that  all  terms  do  not  need  to  be 
retained  for  the  classification  process,  and  computationally  it  would 
be  impractical  to  do  so.  Therefore,  a  word  selection  measure  is  em¬ 
ployed  to  delete  those  terms  that  rarely  occur  and  those  that  have  a 
low  conditional  probability  of  occurring  in  a  category.  A  set  of  sample 
documents  known  to  belong  to  each  category  is  used  to  estimate  the 
mean  frequency,  the  within  category  variance  and  the  between  category 
variance  of  the  remaining  terms.  These  statistics  are  then  employed 
to  compute  discriminant  functions  which  provide  weighting  coeffi¬ 
cients  for  each  term. 

A  new  document  is  classified  by  counting  the  frequencies  of  the 
selected  terms  occurring  in  it,  and  weighting  the  difference  between 
this  vector  of  observed  frequencies  and  the  mean  vector  of  every 
category.  The  probability  of  membership  in  each  category  is  com¬ 
puted  and  the  document  is  assigned  to  the  category  having  the  highest 
probability.  For  applications  in  which  assignment  to  one  category  is 
not  desirable,  the  probabilities  can  be  used  to  indicate  multi-category 
assignment. 

A  thesaurus  capability  allows  the  following  types  of  words  to  be 
considered  equivalent:  inflected  words,  compound  words,  and  seman¬ 
tically  similar  words  with  different  orthographic  spellings.  Since  the 
technique  is  based  on  statistical  measures,  it  can  classify  documents 
written  in  any  language  provided  a  sample  set  of  documents  in  that 
language  is  available. 

Experiments  have  been  conducted  on  several  English  data 
bases,  and  a  further  experiment  is  being  conducted  on  a  German  data 
base.  Classification  results  in  a  recent  experiment  have  ranged  from 
73  to  95  percent. 


INTRODUCTION 


Both  indexing  and  classification  accomplish  the  same  process  of 
assigning  a  tag  to  a  document,  and  have  the  same  objective  of  retrieving 
relevant  documents  on  the  basis  of  their  tags.  A  classification  system, 
in  addition  to  providing  tags,  also  provides  an  organization  of  the  tags 
based  on  the  classification  structure.  For  some  applications  assign 
ment  to  a  category  dues  not  provide  a  sufficiently  fine  partition  of  a 
collection  for  effective  retrieval.  Therefore,  we  have  developed  a 
two-stage  technique  consisting  of  searching  for  relevant  categories 
and  then  querying  within  those  categories  for  relevant  documents. 

Classification  of  documents  involves  three  distinct  major 
processes.  The  first  two  processes  of  defining  a  structure  of  cate¬ 
gories  and  determining  a  basis  for  the  classification  decision  are 
usually  performed  by  a  classification^,  while  the  third  process  of 
classifying  documents  into  categories  is  performed  by  a  classifier. 

The  objective  of  our  approach  is  to  develop  computer  techniques  to 
perform  the  second  and  third  processes.  Because  a  particular 
subject  field  may  be  partitioned  in  many  ways  depending  upon  the 
point  of  view  and  needs  of  the  user,  we  believe  that  the  classifi- 
cationist's  first  process  must  be  influenced  by  the  needs  of  his  or¬ 
ganization.  Therefore,  rather  than  attempt  to  define  categories  or 
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cluster  documents  statistically  to  determine  a  mathematically  optimum 
partition,  we  accept  the  user's  structure  and  start  our  technique  with 
a  sample  of  documents  known  to  belong  to  each  category.  Each  cate¬ 
gory  in  the  structure  is  considered  to  ue  a  node  in  a  tree,  and  all 
nodes  below  ihat  node  are  its  subcategories. 

Our  current  computer  programs  perform  the  second  and  third 
processes.  The  first  set  of  programs  attempts  to  detect  a  pattern 
among  the  documents  and  then  select  and  weight  a  subset  of  words  to 
form  a  basis  for  the  classification  decision.  These  classificationist 
programs  are  used  only  when  the  system  is  initiated  or  revised, 
whereas  the  second  sot  of  programs  are  used  periodically  to  classify 
new  documents. 

The  classifier  programs  could  be  modified  to  not  only  classify 
new  documents  but  also  store  frequency  counts  on  all  words  observed 
in  the  documents,  along  with  the  categories  to  which  it  was  assigned. 
Periodically  (or  on  demand),  comparisons  could  then  be  made  between 
statistics  collected  from  the  new  documents  and  the  statistics  collected 
on  the  original  documents.  When  a  significant  difference  occurred  in 
any  one  of  the  statistics,  an  output  could  be  generated  for  perusr  by 
the  classificationist. 
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Information  required  for  the  addition  of  categories  can  be  ob¬ 
tained  readily  be  observing  an  increase  in  the  arrival  rate  of  new 
items.  Information  for  the  deletion  of  categories  can  be  obtained  by 
observing  either  a  decrease  in  the  arrival  rate  of  documents  in  a 
specific  category  or  a  decrease  in  the  arrival  rate  of  terms  in  the 
discriminating  subset.  In  our  technique,  the  categories  are  actually 
defined  by  only  a  small  subset  of  terms.  By  changing  the  terms 
within  the  subset,  the  definition  of  the  categories  will  be  changed. 
Statistics  indicating  the  potential  discriminating  power  and  the 
coverage  of  each  term  will  be  maintained  separately  for  each  cate¬ 
gory.  Thus  the  need  for  creating  an  inter-disciplinary  category  can 
be  observed  when  the  arrival  rate  of  a  term  increases  simultaneously 
in  several  apparently  unrelated  categories. 

CLASSIFICATION  PROCEDURE 

A  user  selects  a  classification  structure  and  a  sample  of  doc¬ 
uments  known  to  belong  to  c->rh  category.  The  text  of  these  docu¬ 
ments  (or  abstracts)  is  entered  into  the  computer.  A  word  fre¬ 
quency  program  counts  the  frequency  of  eai  h  word  type  for  each 
category. 

Previous  experiments  indicate  that  all  word  types  do  not  need 


to  be  retained  for  the  classification  process,  and  computationally  it 


would  be  impractical  to  do  so.  Ideally,  words  selected  to  represent 
the  categories  should  occur  in  one  and  only  one  category.  However, 
there  are  usually  only  a  few  words  in  any  data  base  that  occur  in  one 
and  only  one  category,  and  these  words  do  not  necessarily  occur  in 
every  document.  Therefore,  a  word  selection  statistic  is  needed  to 
identify  words  approximating  this  condition,  and  to  select  a  subset 
of  words  to  form  the  basis  of  the  classification  decision.  The  sta¬ 
tistic  chosen  is  the  log  of  the  ratio  of  the  relative  frequency  of  a 
word  in  a  category  to  the  relative  number  of  documents  in  that  cate¬ 
gory,  and  this  is  computed  for  each  word  in  the  category. 

For  each  word,  the  value  of  this  statistic  is  compared  across 
all  categories,  and  a  particular  word  is  placed  in  the  list  of  that 
category  in  which  it  has  the  most  positive  value.  After  all  words 
have  been  placed  in  a  category,  the  word  list  for  each  category  is 
arranged  in  descending  values  of  the  statistics. 

Finally,  to  represent  each  category  in  the  structure,  words 
are  selected  according  to  two  criteria.  Words  must  not  only  have 
a  high  word  .selection  statistic  value,  but  they  must  also  occur  in 
some  specified  minimum  number  of  documents  in  the  category. 
Thus,  the  latter  criterion  is  needed  to  ensure  that  the  subset  of 
words  selected  will  provide  a  significantly  high  percentage 
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coverage.  The  words  that  satisfy  both  requirements  will  be  called 
discriminating  words.  They  form  the  basis  for  all  classification 
decisions  to  be  made  at  the  branch  point  for  which  they  were 
developed. 

At  present  our  computer  programs  will  accept  only  100  of 
these  discriminating  words.  In  order  to  obtain  maximum  coverage 
from  this  relatively  small  set,  the  following  thesaurus  techniques 
have  been  incorporated: 

(1)  Various  inflections  of  a  word  are  combined 
with  its  root  word 

(2)  Compound  words  are  combined  with  their 
root  words 

(3)  Synonyms  and  related  words  are  tagged  with 
the  same  internal  word  number. 

These  techniques  effected  an  increase  in  coverage  of  approximately 
200  more  words. 

The  sample  set  of  digests  for  each  category  are  again 
processed  to  compute  the  mean  frequency  of  each  discriminating 
word  for  each  category,  the  pooled  within-catcgory  dispersion,  W, 
and  the  among-category  dispersion,  A.  The  optimum  set  of 
weighting  coefficients  is  found  by  solving  the  determinantal 
equation,  Jw“*  A-1\J  =  0,  for  its  eigenvalues,  A  •  The  eigen¬ 
values  are  then  used  to  compute  eigenvectors  whose  elements  arc 


the  desired  weighting  coefficients.  The  number  of  non-zero  eigen¬ 
values  of  the  determinantal  equation  is  at  most  equal  to  the  smaller 
of  the  number  of  categories  minus  one  or  the  number  of  variables. 
Thus,  our  technique  is  independent  of  the  number  of  categories.  If 
a  group  contained  ten  categories,  nine  eigenvalues  would  be  found 
which  would  provide  nine  sets  of  weighting  coefficients  for  each 
word. 

The  eigenvalue  solution  also  provides  the  basis  of  an  ortho¬ 
gonal  discriminant  (classification)  space.  The  eigenvectors  are 
used  to  transform  each  category  mean  and  dispersion  from  the 
original  100-variable  space  to  a  reduced  classification  space. 

A  new  document  is  classified  by  counting  the  frequencies  of 
the  discriminating  words  occurring  in  it,  transforming  this  fre¬ 
quency  vector  to  the  classification  space  (weighting  its  words)  and 
comparing  it  with  the  mean  vector  of  every  category.  The  proba¬ 
bility  of  membership  in  each  category  is  computed  and  the  dot  u- 
ment  is  assigned  (o  the  category  having  the  highest  probability. 

For  applications  in  which  assignment  to  only  one  category  is  not 
desirable,  the  probabilities  for  each  category  may  be  stored  for 
future  retrieval. 
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WORD  FREQUENCY  PROGRAM 


In  conjunction  with  this  project  a  generalized  character  and 
word  frequency  program  has  been  developed  for  the  System/ 360 
computer.  These  programs  (1)  are  being  used  in  the  computer 
classification  experiments  and  can  be  used  independently  for  any 
language  analysis  study  involving  the  statistical  and  morphological 
behavior  of  character  strings  or  items  in  narrative  text. 

The  S/360  program  is  written  in  FORTRAN  IV  and  can  be 
easily  adapted  to  a  360  model  available  to  the  user.  The  program 
provides  numerous  user  options  concerning  the  definition  of  a 
countab'e  item  (e.  g.  ,  a  single  character  or  a  character  string, 
which  may  or  may  not  be  a  word;  a  "word"  may  be  specified  as  any 
string  of  characters  between  delimiters  such  as  comma,  space, 
period,  or  any  combination  thereof),  the  definition  of  the  textual 
units  over  which  frequencies  are  to  be  subtotaled  (c.  g.  ,  sentence, 
paragraph,  and/or  document),  the  types  of  data  to  be  output,  and 
the  machine  eonliguration  to  be  used. 

The  modular  program  design  provides  subroutines  that 
perform  functions  basic  to  -11  applications  and  subroutines  that 
perform  optional  functions  specified  by  the  user.  It  also  all*  >  w  b 
for  the  incorporation  of  new  programs  to  be  written  by  the  user  to 


perform  additional  optional  functions.  The  basic  subroutines  in¬ 
corporated  in  the  program  perform  the  input  and  item  identification, 
dictionary  building,  merging,  and  frequency  output  functions.  The 
program-provided  optional  subroutines  perform  the  concordance, 
special  item  meek,  summary  output,  growth  rate,  and  detailed 
frequency  print  functions.  Some  user-provided  optional  programs 
could  perform  pre-processing,  interval  definition,  encoding,  word 
use  tagging,  and  special  action  on  specific  word  functions. 

Detailed  output  available  for  each  item  includes  the  item 
itself,  its  character  length,  its  frequency  in  absolute  and  per¬ 
centage  form,  the  location  its  first  occurrence  and  the  number 
of  textual  units  in  which  it  appeared.  Summary  outputs  available 
are  vocabulary  growth  rate,  distribution,  item  types  by  initial 
character,  item  types  by  s  ring  ongth,  item  tokens  by  string 
length,  and  a  concordance  of  Items,  tags,  interval  identification 
and  sequential  position  within  interval.  Each  of  these  outputs  may 
be  obtained  for  any  or  all  textual  units. 

CLASSIFICATION  EXPERIMENTS 

A  series  of  experiments  have  been  conducted  to  demonstrate 
the  generality  of  the  technique  on  data  bases  from  various  dis¬ 
ciplines  and  on  data  bases  in  the  English  and  German  languages. 
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The  experiments  have  also  provided  information  on  ranges  of  values 
of  significant  parameters,  which  are  necessary  to  determine  the 
effectiveness  of  the  technique  on  a  particular  data  base. 

Table  1  contains  a  summary  of  the  results  and  conditions  of 
four  experiments.  The  earliest  work  (2)  consisted  of  a  computer 
evaluation  of  the  form  of  the  classification  equations  proposed  by 
Edmondson  and  Wyllys  (3)  and  classification  experiments  on  corn- 
puter  abstracts  of  the  same  type  used  by  Maron  (4)  and  Borko  (5). 
These  experiments  (6)  indicated  that  better  results  couid  be  achieved 
by  using  a  subset  of  all  the  words  occurring  in  a  document  collection 
and  by  weighting  words  according  to  their  discrimination  ability  rather 
than  treating  each  word  equally  in  the  classification  decision. 

Many  statistical  techniques  exist  for  the  classification  of  a 
random  observation  into  one  of  two  populations.  However,  not  until 
recently  have  techniques  been  developed  for  classifying  observations 
into  many  categories.  A  survey  of  the  techniques  has  indicated  that 
multiple  discriminant  functions  appear  to  be  the  best  statistical 
technique  for  document  classification.  The  functions  not  only  provide 
weighting  coefficients  that  reflect  a  word's  discriminating  ability  but 
they  also  offer  the  optimum  classification  decision  rule  (7)  when  the 
multivariate  data  is  normally  distributed.  Data  from  the  solid  state 
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Table  1.  Summary  of  Classification  Experiments 


Subject  Field 

Computer 

Legal 

Solid  State 

Computer 

Language 

English 

English 

English 

German 

Type  of  document 

Abstract 

Document 

Abstract 

Abstract 

%  agreement  of  computer 
with  original  classifi¬ 
cation 

Sample  documents 

— 

98% 

88% 

94% 

Test  documents 

67% 

74% 

79% 

90% 

Source  of  original 

ccc* 

West 

ccc* 

GFRPO** 

classifications 

#  Documents  available 

400 

5000 

2754 

5000 

#  Documents  included  in 

400 

885 

1743 

2097 

experimental  structure 

#  Sample  doc-nents  in 

15,  75 

20-48 

35,  70,  140 

141-937 

each  category 

#  Levels  in  experimen- 

2 

2 

2 

3 

tal  structure 

#  Groups  in  experimen- 

5 

2 

2 

3 

tal  structure 

#  Categories  in  a  group 

4,  5 

4,  5 

3 

2,  3 

Total  #  of  categories 

24 

9 

6 

7 

#  Discriminating  words 

20 

48 

48 

100 

Average  length  of 

90 

1000 

90 

30 

document 

Average  #  of  discrimi- 

-  ~ 

10 

6 

3 

nating  words  in  docu¬ 
ment 

Thesaurus  capability 

No 

No 

Yes 

Yes 

*CCC  is  Cambridge  Communication  Corporation. 

**GFRPO  is  German  Federal  Republic  Patent  Office 
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experiment  plotted  in  Figure  1  indicates  that  the  coordinates  of  docu¬ 
ments  in  the  classification  space  appear  to  be  bivariate  normally 
distributed  since  they  are  enclosed  by  an  ellipse.  The  data  in  the 
upper  plot  is  based  on  the  sample  documents  used  to  generate  the 
system  whereas  the  lower  plot  consists  of  new  documents  that  are 
presented  to  the  system  for  classification.  An  ellipse  indicating  the 
99%  contour  line  should  encase  the  observations  of  a  sample  or  pop¬ 
ulation  with  a  99%  probability.  Since  the  plot  of  sample  documents  is 
similar  to  the  plot  of  an  independent  set  it  has  been  concluded  that 
the  distribution  of  the  sample  is  an  adequate  representation  of  the 
population  distribution,  and  they  are  both  normally  distributed. 

Multiple  discriminant  functions  have  been  used  in  each  of  the 
succeeding  experiments.  The  legal  experiment  demonstrated  that 
documents  longer  than  abstracts  could  be  handled.  The  documents 
ranged  in  length  from  500  to  5000  words.  The  longer  documents 
performed  better  than  the  shorter  ones.  The  legal  profession 
requires  two  different  types  of  searches  on  the  same  data  base. 

The  /  may  wish  to  find  a  document  relevant  to  points  of  law  in  the 
case  at  hand  or  they  may  wish  to  find  a  document  relevant  to  the 
facts  in  the  case  at  hand.  Thus,  the  same  data  base  must  be  par¬ 
titioned  and  classified  from  two  points  of  view.  This  was 
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•  rrtplished  by  first  selecting  a  subset  of  law  words  and  a  subset  of 
fact  words,  and  secondly,  classifying  each  document  twice.  The  two 
resulting  files  are  independent  and  searches  may  be  addressed  to  either 
or  both  files.  No  significant  difference  was  observed  in  classification 
performance  between  .he  two  classification  systems. 

The  solid  state  experiment  (8)  provided  information  on  the 
significant  parameters  affecting  classification  performance.  The 
parameters  studied  were  the  number  of  sample  documents  required 
to  define  a  category,  the  length  of  documents,  the  interrelationships 
of  the  number  of  sample  documents  and  their  lengths,  the  relation  of 
the  number  of  word  types  in  a  document  to  the  number  of  categories 
assigned  to  it,  levels  in  a  structure,  homogeneity  of  categories,  and 
the  number  of  discriminating  types  occurring  in  a  document. 

The  number  of  sample  documents  required  to  form  the  basis  of 
the  classification  decision  appeared  to  be  an  important  parameter. 
Experiments  were  conducted  with  35,  70,  and  140  sample  documents 
per  category.  As  the  number  of  sample  doc  uments  increased  the 
performance  on  the  sample  decreased  whereas  the  performance  on 
the  independent  test  set  increased.  When  performance  on  both  sets 
converge,  the  maximum  performance  of  the  system  can  be  determined 
(if  no  other  parameters  are  changed)  and  it  can  be  concluded  that  the 
sample  is  representative  of  the  population. 


Performance  is  not  wholly  dependent  on  the  number  of  sample 
documents  but  rather  on  the  total  words  in  the  sample.  Thus,  fewer 
longer  documents  may  be  required  to  reach  a  stable  point  as  in  the 
legal  experiment  where  as  few  as  twenty  sample  documents  were  used 
having  a  length  of  500  to  5000  words.  The  difference  between  the  solid 
state  sample  and  the  test  results  is  much  less  than  the  difference  in 
the  legal  results  as  shown  in  Table  1. 

The  classification  procedure  in  a  structure  consisting  of  many 
levels  and  many  subcategories  involves  an  independent  decision  at 
each  branch  point  (node)  in  the  structure.  For  a  structure  containing 
five  levels,  five  classification  decisions  are  made.  The  basis  for  a 
decision  at  one  level  is  independent  of  the  basis  at  another  level.  The 
basis  at  each  node  is  determined  by  the  sample  documents  within  that 
node  and  the  discriminating  subset  of  words  derived  from  those  docu¬ 
ments.  A  different  discriminating  subset  is  used  at  each  node.  Words 
may  or  r,  ay  not  be  members  of  subsets  at  various  nodes,  depending 
upon  their  discriminative  ability  at  a  node.  A  solid  state  experiment 
indicated  that  there  was  no  degradrtiv.a  in  performance  at  a  lower 
level  when  the  number  of  sample  documents  was  held  constant. 

The  latest  experiment  was  performed  on  a  set  of  patent  abstracts 
concerning  computer  circuits  supplied  by  the  IBM  Germany  Patent 
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Department.  The  abstracts  written  in  the  German  language,  were 
original!/  classified  by  the  German  Federal  Republic  Patent  Office. 
Samples  of  documents  were  randomly  selected  from  each  category 
to  derive  the  discriminating  word  subsets  and  to  form,  the  basis  of 
the  classification  decision.  To  preserve  the  a  priori  distribution 
of  documents  over  the  categories,  two-thirds  of  the  documents 
available  in  each  category  were  selected  for  the  sample  set.  This 
yielded  a  range  from  141  to  937  documents  per  category,  the  cate¬ 
gories  at  the  lowest  levels  having  the  fewest  documents. 

Language  translation  programs  were  unnecessary  for  the 
technique  to  operate  on  the  German  language  data  base.  The  pro¬ 
grams  compute  statistics  on  the  words  contained  in  the  sample  docu¬ 
ments. 

A  thesaurus  capability  incorporated  with  the  solid  state  ex¬ 
periment  was  expanded  for  the  German  experiment.  As  the  dis¬ 
criminating  words  are  being  selected,  inflected  forms  of  a  word  are 
considered  equivalent  to  its  root  word,  compound  words  occurring 
with  similar  discriminative  power  in  the  same  category  are  con¬ 
sidered  equivalent  (E’NGANG,  E1NGANGSKLEMME,  E1NGANGSSIGNAL, 
EINGANGSIMPULS),  and  words  having  the  umr  discriminative  power 
in  the  same  category  occurring  with  different  orthographic  repre¬ 
sentations  are  considered  equivalent  (FLIP-FLOP,  MULTI-VIBRATOR). 
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Since  a  different  discriminating  word  set  exists  for  each  group, 
the  thesaurus  relationships  hold  only  for  that  group.  This  provides  a 
solution  for  the  arduous  and  paradoxical  task  of  constructing  a  single 
thesaurus  for  a  given  data  base.  It  allows  contextual  relationships 
dependent  on  the  particular  subject  group.  If  the  word  "pitch"  occurs 
in  three  different  groups  it  can  be  related  to  different  words  in  each 
group:  throw  (sports),  level  (music),  tip  (dynamics). 

The  technique  was  tested  at  the  second,  third,  and  fifth  level  of 
detail  in  the  German  patent  structure.  The  fifth  level  consisted  of 
deciding  within  the  pulse  circuitry  group  whether  the  circuit  generated 
pulses,  switched  pulses  or  counted  pulses.  The  overall  performance 
yielded  90%  agreement  with  the  original  categories  for  the  independent 
test  set  and  94%  for  the  sample  set. 

Successful  computer  classification  experiments  have  been  per¬ 
formed  on  four  data  bases  involving  over  5000  documents  in  two 
languages.  The  experiments  have  yielded  considerable  data  on  the 
significant  classification  parameters  which  can  be  used  to  design 
computer  classification  systems  and  improve  their  performance. 
Consideration  has  been  given  to  problems  of  changing  technology  and 
the  need  for  updating  classification  structures,  reclassifying  docu¬ 
ments  and  recognizing  the  arrival  of  new  terms. 
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A  two-stage  searching  technique  consisting  of  searching  for 
relevant  categories  and  searching  for  relevant  documents  within  a 
category  based  on  a  full  text  strategy  is  now  under  development. 
Documents  are  classified  within  a  structure  and  a  concordance  of 
terms  occurring  in  each  document  is  prepared.  A  query  is  pre¬ 
sented  tcj  the  system  in  the  form  of  a  statement  of  the  problem 
written  in  natural  text  approximately  a  paragraph  long.  The  query 
is  classified  into  one  or  more  categories.  Then  a  fine  search  is 
made  with  a  term  oy  term  comparison  of  the  query  and  each  docu¬ 
ment  in  the  category. 
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